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(54) Digital data watermarking 

(57) Digital watemrtarking of data, including image, 
video and audio data, is performed by repeatedly insert- 
ing the watermark into sut>regions or subimages of the 
data. Similarly, the watermark is repeatedly extracted 
from the siA>regions of the data. 
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Description 

The present invention relates to dgrtal watermark- 
ing of data including image, video and multimedia data. 
Specrflcally. the invention relates to the insertion and 
extraction of embedded signals for purposes of water- 
marking, in which the insertion and extraction proce- 
dures are repeatedly applied to subregions of the data. 
When these subregions conrespond to the 8x8 pixel 
blocks used for MPEG and JPEG corrpression and 
decompression, tfie watermarking procedure can t>e 
tightly coupled with these compression algorithms to 
achieve very significant savings in computatk)n. 

The proliferation of digitized mecfia such as image, 
video and multimedia is CTeating a need for a security 
system which facilitates the identification of the source 
of the material. 

Content provkJers, i.e. owners of works in digital 
data form, have a need to embed signals into 
video/image/multimedia data which can subsequently 
be detected by software and or hardware devices for 
purposes of authenticating copyright ownership, control 
and management 

For exarrple, a coded signal might be inserted in 
data to indicate that the data shoukl not be copied. The 
embedded signal should preserve the image f kielity, be 
robust to common signal transformations and resistant 
to tampering. In addition, conskieration must be given to 
the data rate that can be provided by the system, 
though current rec^irements are relatively low - a few 
bits per frame. 

In EP-A-766 468 corresponding to U.S. Patent 
Application Serial No. 08/534,894. filed September 28, 
1995, entitled "Secure Spread Spectrum Watermarking 
for Multimedia Data' and assigned to the. same 
assignee as tie present inverrtion. which is incorporated 
herein by reference, there was proposed a spread spec- 
trum watermarking method which embedded a water- 
mark signal into perceptually significant regk>rY& of an 
image for the purposes of kientifying the content owner 
and/or possessor. A strength of this approach is that the 
watermark is very difficult to remove. In fact, this method 
only allows the watermark to be read if the original 
image or data is available for conparison. This is 
because the original spectrum of the watermark is 
shaped to that of the image through a non-linear niulti- 
plicative procedure and this spectral shaping must be 
removed prior to detectbn by matched filtering and the 
watermark ts inserted into the N largest spectral coeffi- 
cients, the ranking of whk;h is not preserved after water- 
marking. Thus, this method does not alk>w software and 
hardware devices to directiy read embedded signals. 

In an article by Cox et al., entitied " Secured Spec- 
trum Watermarking for Multimedia" available at 
hittpyAvww.neci.nj.com/tr/index.html (Technical Report 
No. 95-10) spread spectrum watermarking is described 
which embeds a pseudo-random noise sequence into 
the digital data for watermarking purposes. 



The above prior art watermark extraction methodol- 
ogy requires the original image spectrum be subtracted 
from the mtermark image spectrum. This restricts the 
use of ttie method when there is no original image or 
5 original image spectrum available. One application 
where this presents a significant diffKulty is for third 
party devrce provklers desiring to read embedded infor- 
mation for operation or denying operation of such a 
device. 

10 In U.S Patent No. 5,319.735 by R.D. Preuss et al 
entitied "Embedded Signalling" digital information is 
encoded to produce a sequence of code symt)ols. The 
sequence of code symbols is embedded in an audio sig- 
nal by generating a con^esponding sequence of spread 

15 spectrum code signals representing the sequence of 
code symbols. The frequency components of the code 
signal t>eing essentially confined to a preselected sign- 
aling band lying within the bandwidtii of the audb signal 
and successive segments of ttie code signal corre- 

20 spends to successive code symbols in the sequence. 
The audio signal is continuously frequency analyzed 
over a frequency band encompassing the signalling 
band and the code signal is dynamically filtered as a 
function of the analysis to provkie a modified code sig- 

25 nal with frec^ency component levels whk;h are, at each 
time instant, essentially a preselected proportion of the 
levels of the audio signal frequency components in cor- 
responding frequency ranges. The modified code signal 
and the audk) signal are corTt>ined to provide a compos- 

30 rte audio signal in which the digital information is 
erTi>edded. This component audio signal is then 
recorded on a recording medium or is otherwise sub- 
jected to a transmission channel. Two k^ elements of 
th^ process are the spectral shaping and spectral 

35 equalization that occur at the insertion and extraction 
stages, respectively, therek>y allowing the embedded 
signal to t>e extracted without access to the unwater- 
marked original data. 

In EP-patent application No. 97 11 53 53.1 cone- 

40 spending to US. Patent Application Serial No. 
08^08,331 , fned September 4, 1996. entitied "A Spread 
Spectrum Watermark for Embedded Signaling" by Cox, 
and incorporated herein by reference, there is 
descrit>ed a method for extracting a watermark of 

45 erTt>edded data from watermarked images or video 
witiwut using an original or unwatermarked version of 
the data. This work can t>e viewed as an extension of 
the original work of Preuss et al from the audio domain 
to images and vkleo. 

50 This method of watermarking an image or image 
data for embedding signaling requires that the DCT 
(discrete cosine transform) aixJ its inverse of the entire 
image be computed. There are fast algorithms for com- 
puting the DCT in /V log A/ time, where N is the number 

55 of pixels in the image. Ktowever, for /V=512x512, the 
computational requirement is still high, particulariy if the 
encoding and extracting processes must occur at video 
rates, i.e. 30 frames per second. This metiiod requires 
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approximately 30 times ttie computation needed for 
MPEG-II decompression. 

One possible way to achieve real-time video water- 
marking is to only watermark every frame, htawever, 
content owners wish to protect each and every video 
frame. Moreover, rf rt is known which frames contain 
embedded signals, it is simple to remove those frames 
with no noticeable degradation in the vkileo signal. 

In U.S. Patent Application Serial Na 08/715.953. 
filed September 19, 1996. entitled "Watermarking of 
Image Data Using MPEG/SPEG Coefficients" by Cox. 
arxi incorporated herein by reference, there Is 
descrbed an alternative method, which is to insert the 
watermark into n x n bkx:ks of the image (sii)images) 
where n«N Then the corrputation cost is 

N 

-n\oQn=N\oQn. 



For N=512x512^2^^ and n=8x8=2^, the asynp- 
totic saving is only a factor of 3. However, empincally the 
cost of computing the DCT over the entire image may 
be significantly higher when cache, loop unfolding and 
other efficiency Issues are oonskJered. Thus, the practi- 
cal difference may approach a 30 fokJ savings. More 
importantly, if the block size is chosen to be 8x8. i.a the 
same size as that used for MPEG image compression, 
then it is possft)le to tightly couple the watermark inser- 
tion and extraction procedures to those of the MPEG 
compressk>n and decorrpression algorithms. Consider- 
able computational saving can then be achieved since 
the most expenses computations relate to the cateula- 
tion of the DCT and its inverse and these steps are 
already oorrputed as part of the conpression and 
decorTpressk>n algorithm. The incremental cost of 
watermarking is then very small, typically less than 5% 
of the computational requirements associated with 
MPEG. 

The present invention Improves the reliability of the 
inventk>n described in the 08/715,953 applk;ation by 
storing watermark information into sutjimages. and 
extracting watermark information from subimages. in a 
manner different from that descrit}ed earlier. 

The present invention improves the reliability of the 
prior systems by systematically varying the order in 
which watermark signal components are inserted into 
each sut)image. by inserting only part of the watermark 
signal into each subimage. and. during watermark 
detection, by combining the watermark signals found in 
groups of sutxmages to reconstruct the original water- 
mark signal before testing for correlation with any pre- 
defined watermarks 

For detection, a reverse transformation is applied to 
each subimage to reconstruct the watermark informa- 
tion tfiat was stored in that subimaga The resultir)g sig- 
nals are then averaged together to reconstruct the 
whole watermark, and to reduce noisa Rnally, this 



reconstructed watermaric is corrpared against a prede- 
fined set of watermark signals to determine which one 
was inserted into the image. 

A principal object of the present invention is there- 

5 fore, the provision of inserting a sut>set of a watermark 
into a sut>set of subregions of data to be watermarked. 

Another object of the invention is the provision of a 
digital watermarking system in which a watermark is 
extracted by averaging the watermarked signal from 

10 subregions of watermarked data, arxi then correlating 
the resulting signal to determine the watermark. 

A further object of the invention is the provisk>n of a 
digital watermarking system in which the watermark is 
composed of two portions, a verification portion and a 

IS synchronization portion, in order to irrprove watermark 
extraction reliat}ilrty 

Further and still other objects of the invention will 
become more clearly apparent when the following 
description is read in conjunction with the accorrpany- 

20 ing drawing. 

Brief Description of the Drawings 

Rgure 1 is a schematk; block diagram of watermark 

25 insertk>n procedure; 

Rgure 2 is a schematic bkx^k diagram of a water- 
waxk insertion procedure in accordance with the 
teachings of the present inventk>n; 
Rgure 3 is a schematic btock diagram of a water- 

30 mark extraction procedure; 

Rgure 4 is a schematic bkx;k diagram of a water- 
mark extraction procedure in aocordarx;e with the 
teachings of the present inventk>n; 
Rgure 5 is a graphic representation of a zigzag pat- 

35 tern useful for vectorizing subimages; ^ 

Rgure 6 is a ^aphk: representation of rotation of 
PN sequences; 

Rgure 7 is a graphical representation of an 8x8 
block shown the spatial relation of averaged terms; 
40 Rgure 8 is a schematic tAodk diagram of a method 
for inserting watermari® in accordance with the 
present invention; and 

Rgure 9 is a schematic t)lock dia^am of a method 
for extracting watermarks in accordance with the 
45 present invention. 

Petaiied Description 

Referring now to the figures, arxj Rgures 1 through 
50 4 in particular, there is shown schematic bk)ck diagrams 
of a general metfKxi for inserting and detecting water- 
marks in digital data, for instance images. 

In the following description, reference may be made 
to image data or images. While the invention has appli- 
55 cabiiity to image data and images, it will be understood 
that the teachings herein arxi the invention itsetf are 
equally applicat)le to vkieo, image arxi multimeda data 
and the term "image" and "image data" will be under- 
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stood to include these terms where applicable. As used 
herein, "watermark" will be understood to include 
enrt>edded data, symbols, images, instructions or any 
other identifying information. 

In the following desaiption, reference is made to 
procedures descn'bed in U.S. Patent Application Serial 
No. 08/534.894 for inserting and extracting or detecting 
a watermark in Invages as INSERT-ORIGINAL and 
EXTRACT-ORIGINAL, respectively. Reference is made 
to procedures described in U.S. Patent Application 
Serial No. 08^08,331 filed September 4, 1996 for 
inserting and extracting or detecting watermarks in 
images as INSERT-WHOLE and EXTRACT-WHOLE, 
respectively. And reference Is made to procedures 
desaibed in U.S. Patent Application Serial Na 
08/715,953 for inserting and extracting or detecting 
watermarks in images as INSERT-MPEG-A and 
EXTRACT-MPEG-A, respectively. 

Rgure 1 shows a schematic tkxk diagram of 
INSERT-WHOLE procedure for inserting watermarks 
into images. The watermark signal, in the form of a finite 
sequence of symbols chosen from an alphabet, is pro- 
vided as an Input to an error correctkxi encoder 10 
whk;h transforms this sequence into another sec^ence 
that contains redundant information. Hie output of 
erKXXier 1 0 is prcvkied to a PN-mapper 1 1 , whk;h maps 
each symbol of the erKXxJed watermark into a pre-spec- 
if ied pseudo-random noise (Pisl) code. The output of the 
PN-mapper 11 is provided to a spectral transformer 12, 
which converts the pseudo-random noise sequence into 
the frequency domain. The conversion preferably is by 
discrete cosine transfomi (DOT), however, fast fourier 
transform, wavelet type decomposition and the like may 
also be used for frequency conversion. Concun-ently, 
the data to be watermarked is provided to another spec- 
tral transfornrter 13. The outputs of the two spectral 
transformers 12 and 13 are then prcvkied as inputs to a 
spectral shaper 14, which modifies the spectral proper- 
ties of the pseudo-random noise codes from spectral 
transfbrnfYer 12 to mask the watermark when added to 
the image data The spectrally transformed data to be 
watermarked, from spectral transformer 13, is also pro- 
vided as an input to a delay 1 5. The output of the spec- 
tral shaper 14 is then added to the output of delay 15 at 
a surrvner 16. The summer output is subject to an 
inverse transform 1 7. The result of the inverse transform 
is watermarked data. 

INSERT-MPEG-A drffers from INSERT-WHOLE by 
segmenting the data to be watermarked into multiple 
blocks, such as 8x8 pixel sut>images or subregbns. 
Each bkx^k of data then has the watermark inserted 
according to the ahovB 6escnbed method. That is. for 
each 8x8 subimage or subregion, a pseudo-rarxtom 
number (PN) sequence is inserted into the DCT coeffi- 
cients after suitable spectral shaping. The procedure Is 
repeated for all such subtmages or subregions. The size 
of the sut>image or subregion is preferably 8x8, but it 
can be of other sizes, such as 2x2, 3x3, 4x4 or 16x16. 



Rgure 2 shows a schematic block diagram of a 
watermark insertion procedure in accordance with 
teachings of the present invention. The watermark sig- 
nal is processed into a noise spectrum signal by the 

5 error correction encoder 20, the PN mapper 21 , and the 
spectral transformer 22, in the same manner as 
descrit>ed in conjunction with Rgure 1. However, unlike 
INSERT-WHOLE or INSERT-MPEG-A, the watermark is 
then used as an input to a watermark segmenter 23, 

10 which systematk;ally separates the watermark into sev- 
eral subwatermarks. Any portk>n of the original water- 
mark might appear redundantly in several of the 
resulting subwatenmarks. Concurrently, the data to be 
waternwked is used as an input to data segmenter 24, 

IS which segments the. data into bk>cks or subregons, 
such as 8x8 subimages, as in INSERT-MPEG-A. Each 
of the subwatermarks output by the watermark seg- 
menter 23 is then inserted into a data bkx;k by one of 
the watermaric inserters 25a, 25b, etc. The procedure 

20 used by the watermark inserters 25a, 25b, etc., is the 
same procedure described connection with watermari^ 
inserter 18 in Rgure 1. That is, each subwatermark is 
added into a spectrally transformed data block after 
spectral shaping, and the resulting data is then trans- 

25 formed back into the spatial domain. Rnally, the water- 
marked data bkx^ks from the watermark ir>serters 25a, 
25b, etc., are assembled by data combiner 26 to pro- 
duce watermarked data 

Rgure 3 shows a schematic bkx:k diagram of the 

30 EXTRACT-WHOLE procedure. The walemnarked 
image, video or multimedia data is first used as input 
into a spectral normalizer 30 to undo any previously per- 
formed spectral shaping. If the data contains a water- 
mark, then the output of the spectral normalizer 30 will 

35 resemble the.spectral transformation of the PN coding 
of that watermark (the signal that was input to the spec- 
tral shaper 14 in Rgure 1). The output of the spectral 
normalizer 30 is then used as an input to several conre- 
lators 31a, 31b. etc.. which test the watermaric with the 

40 PN codes used to represent the various symbols ttiat 
the encoded watermark might contain (i.a each correla- 
tor tests for one PN code that is used to encode a sym- 
tx)l t)y the PN niapper 1 1 of Rgure 1 ). The outputs of the 
correlators 31a. 31b, etc., are used as inputs to a ded- 

45 sion circuit 32, which determines the most likely 
sequence of symbols. Rnally, this sequence is cor- 
rected by an, error corrector 33, which performs the 
inverse coding that was performed by the erra confec- 
tion encoder 10 in Rgure 1. The result is the extracted 

50 watermark. 

In EXTRACT-MPEG-A, the data from which a 
watermark is to be extracted is first segmented into sev- 
eral bkx:ks, such as 8x8 subimages, exactly as in 
INSERT-MPEG-A. The signal from each subimage is 

55 then normalized and used as input into a t>ank of corre- 
lators similar to the conrelators 31a, 31b, etc. in Rgure 
3. The output from the correlators is then averaged with 
the outputs of corresponding correlators from other 
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subimages. and the resulting average con-elations are 
used as inputs into the decision drcurt 32 for subse- 
quent processing as described abcva 

Figure 4 shows a scheniatic block diagram of a 
watermark extraction procedure in accordance with the 
teachings of the present Invention. The watermarked 
data is first segmented Into bkxks by data segmenter 
40, which corresponds to the data segmenter 24 used 
during the insertion procedure in Rgure 2. Each of the 
data blocks Is provided to a respective spectrum nor- 
malizer 41a, 41 bi etc. to produce a signal resembling 
the subwatermark that was Inserted Into the respective 
data block. These inserted subwatermark signals are 
then used as inputs into a watermark combiner 42. In 
the combiner 42, parts of the watermark that appear 
redundantly In several subwatermarks are averaged 
together to reduce noise. The output of the watermark 
conrt)iner 42 is provided as the Input to a symbol sepa- 
rator 43 which divides the watermark into parts, each of 
which corresponds to one symbol from the encoded 
watermark signal (the output of error correction encoder 
20 in Rgure 2). 

These syrTtx)ls from separator 43 are provided as 
inputs to respective waternrtark identifiers 44a, 44b etc. 
each of which includes of a bank of con^elators and a 
decision circuit as shown in Rgure 3. The outputs of the 
watermark identifiers are syrTtx)ls from the alphabet 
used in the original encoded watenmark signal. The 
identified symbols are reassembled into a complete 
encoded watenmark by the symbol combiner 45. Rnally, 
the resulting encoded watermark is decoded by the 
error corrector 46. 

The insertk>n and extraction procedures will now be 
desabed in more detail. In INSEFTT-ORIGINAL and 
.EXTRACT-ORIGINAL, the object is to embed a single 
PN (pseudo random number) sequence into an image 
when the original image is available at the time of 
extraction. The information associated with the PN 
sequence is assumed to be stored in a database 
together with the origpnal image and the spectral loca- 
tion of the embedded watermark. The locations of the 
watemnarked components has to be recorded because 
the irrplementation approximated the N perceptually 
vnosX significant regions of the watermark by the N larg- 
est coefficients. However, this ranking was not Invariant 
to the watermarking process. The N largest coefficients 
may be cfiffer^ after inserting the watermark than 
t>ef6re inserting the watennark. 

In order to avoid this problem, the present invention 
places a watermark In predetermined kx;ations of the 
spectrum, typically the first N coefficients. However, any 
predetermined kx»tions coukJ be used, though such 
locations sfiouM t>ek>ng to the perceptually significant 
regions of the spectrum if the watermark is to survive 
comnrK>n signals transformations such as oonrpression, 
scaling, etc. 

More generally, the information to be embedded is 
a sequence of m symbols drawn from an alphabet A 



(e.g. the binary digits or the ASCII symbols). This data 
is tf^ supplemented with additk>nal symbols for error 
detection and correction. Each symbol is then spread 
spectrum modulated, a process that maps each symbol 

5 into a unique PN sequence known as a chip. The 
number of bits per chip is preset - the tonger the chp 
length, the higher the detected signal-to-nolse ratio will 
be. but th^ Is at the expense of signaling bandwidth. 
The power spectrum of the PN sequence is white, 

10 i.e. flat, and Is therefore shaped to match that of the 
"notse". I.e. the imageAndeo/audic/or multimedia data 
into which the watenmark Is to be embedded. It is this 
spectral shaping that must t>e modified from the prior 
methods so that the extraction process no k>nger 

15 requires the original image. To do this, each coefficient 
of the watermarked spectrum is scaled by the kx^al 
average of the power in the image spectral coefficient 
rather than the coefficient itself, i.e. 

20 ff=ff + aavg{\ff\iWf (1) 

The averaging is the averaging of the at>solute coef- 
f k^ient values and not the coeff k»ent values themselves. 
This is effectively estimating the average power present 

25 at each frequency Other averaging procedures are pos- 
sible, for example, averaging over several frames or 
average of local neighborhoods of 8x8 bk>cks. 

This average may be obtained in several ways. It 
may be a local average over a two dimensional region. 

30 Alternative, the two dimensional spectrum may t>e 
sampled to form a one dimensk>nal vector and a one 
dimensional local average may be performed. One 
dlmensKHial vectorizatk>n of the two dimensional 8x8 
DCT coeffkaents Is already performed as part of MPEG 

35 Il.The average nnay t}e a simple box or weighted aver- 
age ever the neighborhood. 

For video data, temporal averacpng of the spectral 
coefficients over several frames can also t>e applied. 
However, since several frames are needed for averag- 

40 ing at the spectral rxxmalization stage of the extractor, 
the protection of individual vkieo frames taken In isola- 
tion may not be possible. For this reason, the present 
inventbn treats vkieo as a very large collection of still 
images. In this way, even Individual vkieo frames are 

45 copy protected. 

In order to extract the watermark, it is necessary to 
perform the spectral normalization, in which the previ- 
ously performed spectral shaping procedure Is inverted. 
In the present Invention, the original unwatermarked 

50 signal is not availak)le. Thus, the average power of the 
frequency ooeffk;ients, avg[\fi\iis approximated by the 
average of the watermarked signal, i.e. avg{\ff\) 

av^5(KyD « svgdfid (2) 

55 

This is approximately true since aaMgf(|//|)l4^-«^, 
where Wj Is the watermark component, and a is a con- 
stant typically In the range between 0.1 and 0.01 . 
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The normalization stage then divides each coeffi- 
cient (f 'i ) in the received signal by the local average avg 
in the neighkx>rhood. 
That is, 



+aWf (3) 



The first term, on the right hand side (RHS) of io 
Equation (3) , 



15 



is considered a noise term. This term was not present in 
the system desCTit>ed U.S. Patent Application Serial htoi 
08/534,894, because access to the unwatermarked 20 
coefficients allowed this term to be removed. The sec- 
ond X&m a Wj is the original watermark signal which can 
now k>e detected using conventional correlation. 

K the watermark is extracted from any single 8x8 
block, the detector reliability is very low. If, however, the 25 
watermarks extracted from each 8x8 bkxk are first 
added together arxi the averaged watermark is then 
applied to the correlator, then a very strong arxi unam- 
biguous response is obtained. This differs from the 
method deserved in U.S. Patent Application Serial l4o. 30 
08/715,953 in which correlation occurred within each 
block and the output from each correlator was averaged 
together. The present invention was found to improve 
the detection response and significantly reduced the 
computation requirement associated with each t)kKk. . as 

In practidng the present invention preferably there 
is a unique PN sequence for each symbol in the alpha- 
bet The method is relatively robust to clipping since the 
detector output reduces linearly with the quantity of 8x8 
sutxmage blocks in the image. For DVD (cfigital video 40 
disk) embedded signaling for APS (anak)g protection 
system) and CCMS (copy generatk>n management sys- 
tem), there woukJ be a total of 8 or 1 6 PN sequences. 

The number of 8x8 blocks in a 512x512 image is 
4096, suggesting that significantly more than one of 1 6 45 
symbols can be embedded in an image or video frame. 
Assume, for example, that it is desired to embed 1 out of 
128 symbols in an image. It is necessary to periorm 1 28 
parallel con-elations. This is computatbnally tractable 
but hardware irrplemenlations of each correlation so 
become wore complex. An alternative method is to only 
use two binary synrtx>ls. It may be preferable to associ- 
ate more than one PN sequence with each of the two 
binary symbols or bits in order to increase the difficulty 
of intentionally removing the watermark. In this case, ss 
there are only two correlators and a binary string may 
be embedded into the image. The raw tit error rate will 
be very high due by tiie low detector output However, 



this can be reduced to acceptable levels by using error 
correcting codes, such as Reed-Solomon (RS). RS 
codes are rokxjst to burst error which may occur 
because of clipping of the image. Other error correcting 
codes may also be used. 

When using ttiis method, it is necessary for tiie 
receiver to know tiie start location of ttie encoded block. 
The start location may not be obvious, particularly when 
tiie image has been subjected to clipping. However, 
convention synchronizing methods can be used; such 
as precedng each block witii a special or unique sym- 
bol or string of symbols. 

To insert a watermark, each 8x8 t)lock^ treated as 
an indivkiual subimage or subregion. Tile OCT of tiie 
subimage is then computed and the two dimensional 
DOT is vectorized in the zigzag pattern shown in Rgure 
5, atttiough other patterns are also possQDie. These two 
stages constitute nrK>st of the calculations but are part of 
the MPEG encoding process Next, a PN noise 
sequence {W| ...W,,} is inserted into the DCT coeffk^tents 
using Equation 1 as before. The lengtti of tiie PN 
sequence cannot exceed 64 (in an 8 x 8 block) and is 
typically much shorter, in the range of 1 1 to 25. If only a 
single code is to be inserted into the image, then the 
same PN sequence is inserted into each of the 720 x 
480/64 - 5400 blocks. However, a variation may be per- 
formed at tiiis point in ttie procedure. Within each row of 
bkx:ks. the PN sequence is cydk^alty rotated by one fre- 
quency coeffKient prior to insertion in tiie suk>sequent 
bkx:k. Similariy. the PN sequence is cydk^ally rotated by 
one frequency coefficient at the start of each new row. 
Rgure 6 illustrates an order of rotations. 

The pupose of these rotations or shifts is to 
improve tiie response of the watermark extraction 
staga Eariier experiments revealed that, certain- DOT 
coeff ictenis were more diffkxilt to estimate tfian others. 
The location at these coeffk^ents varied from image to 
image. However, witfiin an im^e. the coefficient couM 
be consistentiy poor. Consequentiy, without shifting, 
one or more of the estimated watermark coefficients 
couM be significantly degraded relative to the other 
watermark coefficients, theret>y reducing tfie detector 
performance. Conversely, shifting signffkantiy reduces 
the effect a poor DCT coefficients has on a single water- 
mark coefficient and the detector peribrmance is mark- 
edly improved, fstote that any cyclic pattern can be used. 

Further modifications are useful once rotation of the 
watermark has been introduced. Rrst, the length of the 
watermark may now be significantiy greater than 64. 
Then, for each bk)ck only a small sut>set of the water- 
mark (say five) coeff idents is inserted into the first five 
DCT coeffidents (exduding tiie d.c. tenn). Because of 
the rotation, a different sut>set of the watennark 
inserted into neighboring 8x8 blocks. Rnally, having 
completed the watermark insertion, the MPEG encoder 
is abUe to proceed with the sut>sequent stages of com- 
pression. 

Note that the watermark may also be inserted after 
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the MPEG quantization stage to reduce distortion of the 
watermark. MPEG-2 perfomns a convenient one dimen- 
sion vectorization called "zigzagging**, which allows a 
simple 3x1 box average to be performed on the coeffi- 
dents (excluding the d.c. term). 

In practice, performance was improved if the aver- 
aging is performed using the 2 four-connected coeffi- 
cients closest to the d.c. term, as illustrated in Rgure 7, 
i.e. the two coefficients above and to the left 

Watermark detection begins by first extracting the 
FN noise sequence from each 8x8 block using Equation 
1. For each blocK the FN sequence is then cyclically 
shifted in the opposite direction by one frequency coef- 
ficients, and the average over all the blocks is then com- 
puted. In practice, this process can be conputed 
incrementally and does not require temporary storage 
of all the extracted watermarks. A weighted averaging 
can also be applied, where the weights are determined 
k>ased on their susceptibility to common signal transfor- 
mations such as low pass filtering. Rnally. the average 
watermark is compared with the original FN sequence 
via correlation. The reason for shifting the watermark in 
the column direction may now t>e apparent If the image 
is clipped on an art)itrary block boundary, then the com- 
puted avmge watermark will simply be rotated by an 
amount that is a function of the relative locatk>n of the 
clipped portion of the imaga Correlatk>n can then be 
performed on all permutations (typk^ally 1 1 to 25) of the 
watermark. The output from the correlator with the max- 
inum value is then used for decision purposes. The 
extraction stage is depicted in Rgure 4. Taking the max- 
imum correlator output over all rotations of the water- 
mark can cause the decision drcurtry to be noisy. To 
improve this, the watermark broken into two pieces; a 
synchronisation portion is of length K and a verification 
portion is N-K. Then, when the watermark is extracted 
as before, correlation is first performed only on all rota- 
tions of the synchronization portion of this watermark. 
The maximum correlation output is noted, tfien tile veri- 
fication portfon of the watermark is rotated by the oorre- 
sponcfing anrxHJnt and a second correlation is performed 
on the verification portions of the watermarks. This 
process significantly improves the overall reliability of 
the system. In the course of experimentation, it was 
noticed that some watermarks performed better than 
others on the same imagery. This was caused by varia- 
tfon in the correlation statistics between the synchroni- 
zation and verifk^atfon portions of the watermark. 
kJeally, the two portions shoukl have very fow correla- 
tions. However, in several cases where watermarks per- 
formed pooriy, it was traced to unexpected correlations 
t>etween the two portions. 

The present invention provides a modification to 
digital watermarking methods in which the original data 
is required for watermark extraction thereby enabling 
watermarking extraction in the ak>sence of an unwater- 
marked or original data. The present invention prefera- 
bly uses MFEG/JFEG coeffidents. An image is divkied 



into typically 8x8 block subimages or sii>regions and 
each subimage Is processed and the results are com- 
bined to derive the extracted watermark. The result is 
extractfon of the watermark with very high conf derx;e. 

5 While the atxive invention desait>es improvements 
to the prior-art INSEITT-WHOLE, INSERT-MFEG-A. 
EXTRACT WHOLE, and EXTRACT-MFEG-A algo- 
rithms, it should be apparent to anyone skilled in the art 
that the same improvements may be applied to any 

10 algorithm for inserting and extracting watermarks in 
image data. This more general view of the present 
invention is shown in Figures 8 and 9. 

Rgure 8 shows a schematic block diagram of the 
general method for inserting watermarks. This general 

75 method makes use of a non-bfock-based watermark 
insertfon algorithm, whk^h shall be referred to hereafter 
as the 'hase insertion algorithm". The watermark 
encoder 80 converts the watermark into a form appro- 
priate for the base insertion algorithm. If the base inser- 

20 tion algorithm is that shown in Rgure 1, for example, 
then the watermark encoder 80 corresponds to the 
watermark encoder 19, which compr^es the error cor- 
rection encoder 1 0. the FN mapper 1 1 . and the spectral 
transformer 12. However, if a different base insertion 

25 algorithm is to be used, then the watermark eruxxier 80 
may perform a different transformation of the water- 
mark. The encoded watermark signal from watermark 
encoder 80 is provkied as an input to watermark seg- 
menter 81, which divties the watermark into a s^ of 

30 subwatermarks. Any portion of the original watermark 
might appear redundantiy in several of the resulting 
sukswatermarks. The data to be watermarked is pro- 
vided as an input to data segmenter 82, which divkJes 
the data into sut>regions. Each subwatermark is 

35 inserted into a. respective data subregion by a water- 
mark inserter 83a, 83t}. etc. The watermark inserters 
implement the base insertion algorithm, so. if tfie base 
insertion algoritiim is that shown in Rgure 1, then each 
watermark inserter 83a, 83b^ etc., oorresporxis to the 

40 watermark inserter 18. whk;h comprises a spectral 
transformer 13, a spectral shaper 14. adelay 15. a sum- 
mer 16, and an inverse transform 17. However, if a dif- 
ferent base insertion algorithm is to be used, then the 
watermark inserters 83a. 83b, eta, may employ a differ- 

45 ent metfiod of inserting subwatermarks into the subre- 
gfons of the data to be watermarked. The outputs from 
the watermaric inserters are assembled in data com- 
biner 84 to provkJe watemnarked data 

Rgure 9 shows a schematk; block diagram of the 

50 corresponding general extraction algorithm. The algo- 
rithm makes use of a l>ase extraction" algorithm that 
corresponds to tiie base insertion algorithm used in 
inserting the watermark into the data to be watermarked 
(Rgure 8). The algorithm in Rgure 9 is sut}stantially ttie 

55 same as the algorithm shewn in Rgure 4. except that in 
the general case, the spectrum normalizers 41a, etc. 
are replaced by watermark extractors 91a etc., which 
implement the base extraction algorithm. That is, if the 
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base insertion algorithm used was the algorithm shown 
in Rgure 1, then the watermark extractors 91a, etc., in 
Rgure 9 will be the spectrum normalizers 41a, etc. in 
Rgure 4. 

While there has been descnbed and illustrated a 
system for inserting a watemrtark into and extracting a 
watermark from watermarked data without using an 
unwatermarked version of the data, it will be apparent to 
those skilled in the art that variatioris and nrKxlifk:ations 
are possit)le without deviating from the broad principles 
and teachings of the present invention which shall be 
limited solely by the scope of the claims appended 
hereto. 

Claims 

1 . A method for inserting a watermark signal into data 
to be watermarked comprising the steps of: 

dividing data to be watermarked into a plurality 
of subregions; 

computing frequency coefficients of the data to 
be watermarked in each sut>region; 
spread spectaim nrxxiulating a watermark sig- 
nal to be Inserted by mapping the watermark 
signal into a PN (pseudo- random noise) 
sequence; 

spectral shaping the PN sequence as a func- 
tion of the average power in each frequency 
coeffk:ient of the data; and 
inserting each spectral shaped PN sec^ence 
into predetermined coeffidents in the data in 
each subregion. 

2. A method for inserting a watermark signal into data 
to be watermarked as set forth in daim 1 , wfiere 
said frequency coeffidents are DOT (discrete 
cosine transform) coefficients. 

3. A method for inserting a watermark signal into data 
to be watermarked as set forth in daim 1 or 2, 
where each subregbn is a 8 x 8 block of pixels, 

4. A method for inserting a watermark signal irrto data 
to bB watermarked as set forth in claim 3,where the 
frequency coeffidents of the watermark signal are 
rotated prior to Inserting of each spectral shaped 
PN sequence into the subregion. 

5. A method for inserting a watermark signal into data 
to be watermarked as set forth in daim 4, where 
only a siteet of the watermark signal frequency 
coeffidents is inserted into any one sut>region. 

6. A method for inserting a watermark signal into data 
to k>e watermarked as set forth in claim 5 , where 
the watermark signal comprises a synchronization 
portion and a verification portion. 



7. A method for inserting a watermark signal into data 
to be watermarked as set forth In any of daims 1 to 
6, where said inserting is performed after the data 
undergoes MPEG quantization processing. 

5 

8. A method for inserting a watermark signal into data 
to be watermarked as set forth in daims 6 or 7. 
where the synchronisation portion and the verifica- 
tion portion have very littie correlation between 

10 each other. 

9. A method for inserting a watermark signal into data 
to be watermarked as set forth in any of daims 1 to 
8. 

IS 

where tiie spectral shaping as a function of the 
average power is typically 3 x 1 window of the 
coeffident obtained from the one-dimensional 
vectorization by zigzagging of twodimension 
20 frequency coeffidents. 

1 0. A method for inserting a watermark signal into data 
to be watermarked as set forth in any of daims 1 to 
8, where the spectral shaping is a function of the 

25 average power based on the two four-connected 
frequency coeff k^ients dosest to the DC term. 

11. A method of extracting a watermark from water- 
marked data comprising the steps of: 

30 

receiving 8uk>regions of watermarked data; 
spectrum normalizing the watermarked data 
from each subregbn to generate respective 
normalized signals; 
35 ....... combining the respective normalized signals 

from each sut>region to generate a single 
watermark; 

correlating the single watermark with predeter- 
mined PN (pseudo-random noise) sequences 

40 corresponding to predetermined symbols to 

prcvkte correlated signals for each predeter- 
mined PN sequence in each subregion; 
dedding whk;h correlated signal is most likely a 
current symtx)l; and extracting a sequence of 

45 most likely current symtx)ls corresponding to 

the watermark 

12. A method of extracting a watermark from water- 
marked data as set forth In daim 1 1 , where the sub- 

50 regions are 8 x 8 blocks used for MPEG encoding 
and decoding. 

13. A method of extracting a watermark from water- 
marked data as set forth in daim 12, where sakJ 

55 combining the normalized signals from each subre- 
gion to generate a single watermark, induding 
removing the relative rotation of the watermark 
between blocks. 
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14. A method of extracting a watermark from water- 
marked data as set forth in claim 13, further com- 
prising sijt>sequentty reconstructing the watermark 
from partial watermarks inserted into each btodk, 

5 

15. A method of extracting a watermark from water- 
marked data as set forth in claim 14, further conn- 
prising weighting the watermark coefficierrts based 
on their kx;ation within the frequerx:y spectrum, 
where the weighting is a function of the susceptibil- io 
ity of each frequency coefficient to common signal 
trartsfformatiorks. 

16. A method of extracting a watermark from water- 
marked data as set forth in daim 11, further conr>- 75 
prising correlating with all rotational shifts of the 
extracted watermark and selecting the maximum 
value. 

17. A method of extracting a watermark from water- 20 
marked data as set forth in daim 11, further com- 
prising correlating with all rotational shifts of a 
synchronization portion of a watermark to deter- 
nrune a maximum value and subsequently rotating a 
verifk;ation portion of the watermark by the same 25 
anrxHjnt as the synchronization portion is rotated to 
obtain the maximum value prior to correlating 
between the verif k:ation portion and predetermined 

PN sequences. 

30 

18. A method of extracting a watermark from water- 
marked data comprising the steps of: 

receiving sut>regions of watermarked data; 
spectrum normalizing the watermarked data 35. 
from each subregion to generate respective 
normalized signals; 

correlating the respective normalized signals 
with predetermined PN sequences corre- 
sponding to predetermined symbols to provide 40 
conrelated signals for each predetermined PN 
sequence in each subregion; 
dedding which conrelated signal is nrK>st likely a 
current symbol in each subregion for provkling 
an extracted symbol stream; 45 
error conrecting the extracted symbol stream; 
and 

extracting a sequence of most likely current 
symbols corresponding to the watermark. 

50 

19. A method of extracting a watermark from water- 
marked data as set fortti in daim 18. where said 
error correction is Reed Solomon enror correction. 

20. A method for inserting a watermark signal Into data ss 
to be watermarked comprising the steps of: 

dividing data to be waternrtarked into a plurality 



of sufc>regions; 

dividing a watermark signal into a plurality of 
subwatermarks where portions of the water- 
mark are contained in more than one sutMvater- 
mark;and 

inserting said plurality of subwatermarks into 
said plurality of subregions. 

21 . A mettKxJ for inserting a watermark signal into data 
to be watennarked as set forth in daim 20, where 
each subwatermark is inserted into a respective 
subregion, so that each subregion contains at least 
one subwatermark. 

22. A method for extracting a watermark signal from 
watermarked data comprisir^g the steps of: 

receiving a plurality of subregions of watermark 
data; 

extracting a subwatermark from each subre- 
gion of said plurality of sid^egions; and 
combining and averaging the subwatermarks 
extracted from all the subregions to obtain a 
signal commensurate with the watermark sig- 
nal. 

23. A method for extracting a watermark signal from 
watermarked data as set forth in daim 22, further 
comprising the steps of: 

dividing the signal commensurate with the 
watermark signal into a plurality of symbol sig- 
nals; 

correlating each symbol signal with a set of 

.predefined signals; ^ 

determining which predefined signal best cor- 
responds to each synrtol signal; and 
combining the best corresponding predeter- 
mined signals to generate the watermark sig- 
nal. 
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